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ABSTRACT 

In  periods  of  declining  budgets  and  downsizing,  it  becomes 
increasingly  important  to  select  the  best  possible  design  and 
development  approaches  that  provide  the  desired  life  cycle  cost 
benefits  while  sustaining  system  capability.  Aging  of  the  United 
States  Air  Force  (USAF)  systems,  factored  with  efforts  to  extend 
their  operational  longevity,  has  an  impact  on  the  systems’  safety 
and  operational  capabilities. 

On  3 December  1 997,  the  commander  of  the  Air  Force  Materiel 
Command  (AFMC)  chartered  an  integrated  product  team  (IPT) 
to  develop  a cost-effective  Air  Force  policy  for  assurance  of 
operational  safety,  suitability,  and  effectiveness  (OSS&E)  of 
USAF  systems.  The  team  developed  this  new  policy  based  on 
proven  commercial  and  U.S.  Government  practices,  processes, 
and  methodologies  in  place  today.  A key  element  of  this  policy 
is  the  certification  process  that  ensures  airworthiness  is  estab- 
lished and  maintained  throughout  the  life  of  the  system.  The 
highlights  of  the  policy  and  the  selected  processes,  best  prac- 
tices, and  methodologies  are  presented. 


L INTRODUCTION 

The  commander  of  the  Air  Force  Materiel  Command  (AFMC), 
General  George  T.  Babbitt,  has  long  been  concerned  about  the 
configuration  control  of  Air  Force  systems.  In  1997,  he  saw  an 
alarming  trend  in  the  mishap  rates  in  fielded  systems  and  end 
items,  which  led  him  to  question  his  technical  staff  as  to  how 
airworthiness  is  managed  within  the  USAF. 

To  address  this  question,  Aeronautical  Systems  Center  (ASC) 
provided  a comprehensive  briefing  in  December  1997.  It  pointed 
out  that  the  USAF  did  not  have  clearly  documented  airworthi- 
ness policies  similar  to  those  that  the  Federal  Aviation 
Administration  (FAA)  imposes  on  commercial  aircraft.  The 
briefing  emphasized  that,  as  a regulatory  body,  the  FAA  is 
empowered  by  the  FAA  Reauthorization  Act  of  1996,  as 
amended.  The  Act  very  clearly  applies  to  civil  aircraft  and  air 
commerce,  but  it  does  not  apply  to  purely  military  aircraft.  If  the 
FAA  were  to  address  military  aircraft,  legislation  would  be 
required.  Further,  the  FAA  would  have  to  expand  its  capabilities 
to  address  military-unique  equipment  and  operations. 

The  briefing  also  points  out  that  airworthiness  is  only  one  ele- 
ment of  overall  flight  safety.  Airworthiness  is  concerned  with 
system  design,  the  quality  of  the  parts  and  their  integration, 
operational  flight  limits,  and  the  maintenance  and  repair  of  the 
aircraft  and  its  equipment  throughout  its  service  life.  The  second 
element  is  the  capability  of  the  aircrew,  which  necessitates 
proper  qualification  and  training  requirements. 

The  safety  of  aircrew  members  is  extremely  important  to  the 
USAF.  Equally  important  is  the  successful  conduct  of  USAF 


military  operations.  Thousands  of  military  and  civilian  lives  are 
at  stake  if  the  outcome  of  a military  operation  cannot  be  con- 
trolled. Recent  experiences  indicate  that  air  dominance  plays  the 
major  role  in  controlling  outcomes.  Therefore,  the  Air  Force’s 
ability  to  accomplish  its  missions  at  will  and  in  a timely  manner 
is  of  the  utmost  importance.  Thus,  the  suitability  of  the  aircraft 
and  its  effectiveness  also  become  key  considerations  along  with 
safety  of  flight.  For  this  reason,  it  was  only  logical  for  General 
Babbitt  to  expand  the  USAF  airworthiness  effort  to  include  the 
operational  safety,  suitability,  & effectiveness  (OSS&E) 
required  for  successful  military  operations. 

The  current  high  rate  of  mishaps,  decreasing  trends  in  the  mis- 
sion capability  rates,  and  the  effectiveness  of  USAF  systems 
have  a profound  effect  on  the  ability  of  the  Air  Force  to  fight 
wars  safely  and  effectively.  These  factors,  combined  with  the 
realities  of  a shrinking  DoD  budget  and  workforce,  demand  that 
our  resource  utilization  be  improved  considerably  to  sustain  our 
air  dominance  in  the  world.  Furthermore,  for  the  past  decade, 
workforce  downsizing  has  constantly  eroded  our  technical  foun- 
dation in  both  experience  and  corporate  knowledge. 

The  USAF  aircraft  mishap  rate  per  100,000  flying  hours  contin- 
ues to  climb.  Mishaps  may  be  caused  by  a variety  of  factors, 
including  human  performance,  weather,  design,  technical  orders/ 
manuals,  operation,  training,  maintenance,  aging  of  systems, 
dwindling  resources,  and  infrastructure.  For  many  USAF  sys- 
tems, aircraft  mishaps  are  largely  caused  by  human  error.  Yet, 
some  of  these  human  error  mishaps  are  attributable  to  inade- 
quacy of  technical  orders  or  manuals,  or  training  of  pilots  or 
maintenance  personnel.  The  USAF  has  the  ability  to  control  the 
factors  that  contribute  to  aircraft  mishaps,  except  purely  human 
errors,  by  improving  technical  rigor  and  applying  disciplined 
engineering  design  techniques  governed  by  appropriate  policies 
and  priorities.  Therefore,  AFMC  has  set  an  objective  by  the  year 
2005  to  reduce  by  half  the  mishap  rate  resulting  from  control- 
lable factors. 

Concerned  about  the  mishap  trends  and  their  potential  impact  on 
national  security,  General  Babbitt  tasked  the  commander  of  the 
Aeronautical  Systems  Center  in  December  1997  to  take  the  lead 
for  the  USAF  in  establishing  a new  policy  to  assure  operational 
safety,  suitability,  and  effectiveness.  The  IPT  chartered  to 
develop  this  policy  faced  several  challenges  that  are  discussed  in 
this  paper.  The  specific  objectives  of  the  IPT  and  the  philosophy 
adopted  to  overcome  those  challenges  are  also  briefly  discussed. 
The  highlights  of  the  policy  are  summarized.  The  major  portion 
of  the  discussion,  however,  is  devoted  to  those  technical  aspects 
that  create  the  bedrock  for  this  proposed  policy.  This  paper 
further  describes  how  those  technical  aspects  could  contribute  to 
reducing  the  mishap  rate.  Additionally,  it  explores  the  costs  for 
applying  the  disciplined  systems  engineering  rigor  and  support- 
ing processes,  best  practices,  and  methodologies. 


Paper  presented  at  the  RTO  AVT  Specialists * Meeting  on  “Design  for  Low  Cost  Operation  and  Support”, 
held  in  Ottawa , Canada , 21-22  October  1999 , and  published  in  RTO  MP-37. 
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2.  DISCUSSION 
IPT  Effort  and  Challenges 

A highly  cohesive,  cross  "Center  of  Excellence"  OSS&E  Inte- 
grated Product  Team  (IPT)  was  formed  to  address  operational 
safety,  suitability,  and  effectiveness  issues  and  develop  options 
for  resolution.  This  effort  affects  USAF  Product  Centers,  Logis- 
tic Centers,  the  Air  Force  Research  Laboratory,  Air  Force  Major 
Commands,  the  Air  Reserve  component,  and  Defense  Logistics 
Agency.  Stakeholders  in  addition  to  the  Air  Force  include  the 
FAA,  Army,  and  Navy.  A tremendous  amount  of  information 
was  collected  from  the  various  services  and  the  FAA.  As  a basis 
for  developing  the  policy,  the  team  conducted  a comprehensive 
review  of  existing  policies,  mishaps,  airworthiness  certifications, 
best  practices,  and  processes.  From  these,  the  team  selected 
information  that  could  support  a disciplined  systems  engineering 
process  for  OSS&E.  In  order  to  avoid  duplication  on  joint  pro- 
grams and  commercial  procurements,  one  of  the  team’s  major 
objectives  was  to  harmonize  with  the  other  services  and  FAA 
the  USAF’s  proposed  approach. 

Mishap  rate  is  a serious  challenge  for  the  USAF.  The  number  of 
fiscal  year  1999  (FY99)  mishaps  has  already  exceeded  the  num- 
ber of  FY98  mishaps."  As  of  February  18,  the  rate  of  major  F-16 
accidents  for  FY99  was  5.83  per  100,000  flying  hours.  The  rate 
in  FY98  was  3.89,  which  was  a 30  percent  increase  over  the 
FY97  rate  of  3.0  and  an  81  percent  increase  over  the  FY96 
record-low  rate  of  2.14  [1].  Engine  failures  and  human  errors 
continue  to  be  the  primary  causes.  However,  most  of  the  aircraft 
are  designed  for  a 20-year  design  life  and  are  flying  today 
beyond  their  service  life  [2].  To  a varying  degree,  all  these 
airplanes  can  be  expected  to  experience  such  aging  problems  as 
cracking  and  corrosion  [3]. 

Furthermore,  the  aging  aircraft  inventory  impacts  safety  and 
creates  economic  burdens.  "Corrosion  and  fatigue  separately 
have  led  to  serious  safety  as  well  as  economic  problems"  [4]. 
These  problems  are  common  to  both  military  and  commercial 
aircraft  and  have  resulted  in  several  mutual  efforts  to  resolve 
those  issues.  For  example,  in  1989,  the  failure  of  the  mainte- 
nance program  to  detect  the  presence  of  fatigue  damage  was 
cited  as  the  probable  cause  for  the  commercial  airline  accident 
[2]  [5]  that  led  to  military  and  commercial  aircraft  policies  and 
priorities  that  benefited  both.  The  FAA  and  NASA  are  develop- 
ing a host  of  advanced,  highly  accurate,  nondestructive 
evaluation  systems  that  will  significantly  improve  the  accuracy 
of  inspections  while  reducing  airframe  disassembly  and  associ- 
ated costs  [6]. 

The  aircraft  aging  problem  has  been  well  recognized  by  the 
USAF  and  is  being  tackled  on  several  fronts,  such  as  structural 
integrity  and  corrosion  control  programs.  The  USAF  has  the 
experience  in  inspection  and  repair  techniques  to  extend  air- 
frame lives  beyond  the  20-year  design  life  [7].  The  USAF 
structural  integrity  program  and  other  services’  efforts,  in 
combination  with  industry  initiatives,  have  kept  aging  issues  in 
check  to  date.  However,  continued  focus  is  needed  in  this  area. 
The  Air  Force  expects  to  mitigate  the  adverse  trend  in  mishap 
rate  through  the  development  and  application  of  the  OSS&E 
policy  and  an  emphasis  on  the  use  of  disciplined  engineering 
and  risk  management  processes. 


Mishap  investigation  reports  provide  much  insight  on  the  pre- 
vention of  mishaps  and  possible  safety  improvements.  The 
analysis  of  the  information  received  reveals  a clear  breakdown 
in  the  Air  Force  technical  processes.  The  technical  processes  are 
inconsistently  applied  across  systems  during  acquisition  and  are 
seldom  applied  during  the  sustainment  phases  of  their  life. 
Because  unauthorized  changes  have  been  made  to  systems  in  the 
field  without  the  application  of  a disciplined  engineering  proc- 
ess, it  is  not  clear  who  is  accountable  or  responsible  for  some 
mishaps.  Thus,  the  second  objective  of  the  team  was  to  require 
the  application  of  a disciplined  engineering  process  throughout 
the  life  of  the  system.  And  a third  objective  was  to  delineate 
clearly  the  roles  and  responsibilities  of  the  organizations  and 
individuals  (that  is,  the  chief  engineer  and  the  single  manager)  in 
the  policy  documents. 

The  IPT  faced  many  challenges  in  creating  the  OSS&E  policy. 
Two  major  challenges  are  especially  difficult  to  overcome  in 
today’s  austere  environment.  The  first  one  is  that  policies  are 
generally  viewed  as  adding  to  the  product’s  cost  unnecessarily. 
The  issuance  of  a policy  requires  coordination  with  a multitude 
of  organizations  with  varying  interests.  Valid  concerns  of  every- 
one should  be  carefully  considered  and  appropriate  adjustments 
made  to  accommodate  those  concerns. 

The  second  major  challenge  is  that  the  acquisition  reform  advo- 
cates view  imposition  of  policy  as  contrary  to  their  initiatives. 
Most  of  the  acquisition  reform  initiatives  enjoy  high  visibility 
within  the  DoD  and  are  resulting  in  significant  cost  savings. 
Therefore,  it  is  extremely  important  to  ensure  that  the  new 
OSS&E  policy  continues  to  embrace  acquisition  reform  initia- 
tives. Open  communications  and  stakeholder  involvement  are 
key  in  facing  both  of  these  challenges  and  gaining  support. 

Weighing  these  factors,  and  recognizing  our  dwindling  re- 
sources, the  team  adopted  a philosophy  to  develop  a policy  that 
fosters  the  combined  use  of  industry  and  Government  resources 
as  a single  team  in  fielding  and  sustaining  capabilities  required 
for  our  national  defense.  The  team  was  cautious  to  avoid  any 
policy  content  that  may  result  in  duplication  of  effort  (e.g., 
obtain  FAA  certification  where  possible)  or  that  may  be  in 
conflict  with  other  initiatives.  For  example,  the  policy  should 
not  restrict  the  use  of  the  cl  ear-accountability-in-design  (CAID) 
approach.  Under  this  approach,  the  contractor  is  given  control  of 
the  design  and  technical  documentation,  while  the  Government 
retains  the  responsibility  for  defining  the  required  performance 
capability.  While  this  eliminates  duplication  of  responsibilities, 
it  also  affords  a contractor  opportunities  to  cut  the  development 
cost  of  products  and  parts.  Given  this  authority  to  control  the 
detailed  design,  technical  data,  manufacturing,  and  quality 
assurance,  in  essence,  contractors  are  provided  the  flexibility  to 
take  advantage  of  nondevelopmental  items  (NDI),  commercial- 
off-the-shelf  (COTS)  products,  and  the  best  practices  & 
processes  they  deem  necessary. 

The  IPT  developed  a common  technical  management  process 
and  created  a draft  guidance  document,  a draft  Air  Force  policy 
directive,  and  a draft  Air  Force  instruction  in  the  short  period  of 
one  year  for  the  Headquarters  AFMC  to  sponsor  at  the  Air  Staff 
level.  In  December  1998,  the  USAF  Chief  of  Staff  directed  the 
cognizant  organizations  to  expedite  formal  coordination  of  this 
policy. 
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USAF  Policy  Highlights 

This  section  provides  the  synopsis  of  the  new  USAF  policy  for 
OSS&E  assurance  that  is  delineated  in  several  hierarchical 
policies  and  related  documents.  The  policy  applies  to  Air  Force 
systems  and  end  items.  The  main  focus  of  the  policy  is  on 
improving  the  technical  disciplines  for  effectively  fielding  Air 
Force  systems.  It  requires  the  Air  Force  to  assure  the  OSS&E  of 
systems  currently  in,  or  entering  the  operational  inventory  and  to 
employ  a disciplined  engineering  process  and  effective  opera- 
tional, training,  supply,  and  maintenance  procedures  to  preserve 
its  OSS&E  throughout  the  operational  life. 

This  policy  mandates: 

1)  Systems  and  end  items  must  be  delivered  with  a baseline 
that  enables  continuing  assurance  of  OSS&E. 

2)  Preservation  of  baseline  OSS&E  characteristics  of  systems 
and  end  items  over  their  operational  life. 

Certifications,  such  as  airworthiness  and  nuclear  surety,  are 
referenced  in  this  policy  as  important,  supporting  processes, 
central  to  baseline  establishment  and  preservation.  These 
focused  activities  remain  stand-alone  Air  Force  policies.  A 
separate  Air  Force  Airworthiness  Certification  policy  has  also 
been  drafted  and  staffed.  Airworthiness  certification  responsi- 
bility resides  with  the  single  manager  (SM)  for  the  program. 

The  single  managers  and  chief  engineers  are  responsible  for 
preservation  of  operational  baselines  and  are  required  to  utilize  a 
disciplined  engineering  process  toward  that  end.  AFMC  further 
assigns  technical  responsibility  to  its  four  Product  Centers.  Each 
Product  Center  is  responsible  for  providing  supporting  policy, 
guidelines,  processes,  and  technical  standards  tailored  to  their 
unique  product  lines.  These  tools  will  assist  the  single  managers 
and  their  chief  engineers  in  accomplishing  their  responsibilities 
and  provide  AFMC  with  necessary  insight  into  the  health  of  the 
fielded  systems  and  end-items.  The  Air  Force  Operating 
Command  will  coordinate  any  changes  to  configuration  or  usage 
with  the  single  manager/chief  engineer. 

While  the  single  managers  have  the  ultimate  responsibility  for 
OSS&E,  they  are  expected  to  delegate  authority  to  competent 
technical  entities,  as  appropriate.  The  new  policy  requires  that  a 
chief  engineer  or  lead  engineer  be  assigned  to  each  program.  It 
allows  the  single  manager  to  delegate  authority  for  OSS&E 
technical  aspects  of  the  SM  responsibilities  to  the  chief  system 
engineer.  As  an  example,  the  chief  engineer  is  responsible  and 
accountable  to  the  SM  for  assessing  the  airworthiness  and  for- 
mulating certification  recommendations  throughout  the  system’s 
operational  life.  He/she  will  be  responsible  for  preserving  the 
baseline  for  fielded  systems.  The  chief  engineer  will  continue  to 
provide  technical  risk  assessments  so  the  single  manager,  in 
conjunction  with  the  users,  can  make  informed  cost  and  per- 
formance tradeoffs. 

Selected  Processes 

The  OSS&E  Process  Overview 

The  OSS&E  process  consists  of  two  parts:  1)  establishing  the 
OSS&E  baseline  and  2)  preserving  OSS&E  baseline  character- 
istics throughout  the  life  of  a system.  To  establish  the  baseline, 
air  systems  are  assessed  for  operational  safety,  suitability,  and 
effectiveness.  To  preserve  the  baseline,  this  OSS&E  assessment 


is  updated  throughout  the  operational  life  as  missions  and 
system  use  change,  or  as  modifications  are  incorporated. 
Correspondingly,  the  airworthiness  certification  process  also 
consists  of  two  parts:  1)  certifying  airworthiness  and  2)  main- 
taining airworthiness  certification  throughout  the  life  of  a 
system.  Aerospace  vehicles  are  initially  certified  for  airworthi- 
ness. To  maintain  airworthiness,  the  certification  is  updated  to 
account  for  configuration  changes  throughout  the  operational 
life  of  the  system. 

Disciplined  Systems  Engineering  Process 

A disciplined  systems  engineering  process  is  a comprehensive, 
orderly,  iterative,  problem-solving  process  that  transforms  vali- 
dated user  needs  into  a cost  effective  product  for  the  customer  to 
use  reliably  throughout  its  defined  life.  Many  supporting  proc- 
esses are  involved  in  transforming  customer’s  requirements  into 
a description  of  a balanced  life-cycle  solution,  including  people, 
products,  and  subprocesses.  The  systems  engineering  process 
and  its  subprocesses  apply  to  new  system  product  and  process 
upgrades  and  modifications,  as  well  as  to  engineering  efforts 
conducted  to  resolve  problems  in  the  field  or  on  the  manufac- 
turing floor. 

The  following  subsections  address  a few  of  the  key  systems 
engineering  subprocesses.  These  are  described  at  a very  top 
level  and  are  not  meant  to  be  all  encompassing.  Some  of  the 
generic  subprocesses  such  as  definition  of  requirements,  incre- 
mental milestones  (that  is,  design  reviews,  etc.),  and  test  and 
integration  are  commonly  applied  across  Air  Force  systems  with 
varying  degrees  of  success.  They  are  necessary  but  are  not 
addressed  in  this  paper.  Refer  to  MIL-HDBK-500,  "Key 
Supplier  Processes  for  Aeronautical  Sector  Acquisition  and 
Sustainment  Programs"  [8]  for  further  detailed  information  on 
those  processes.  The  subprocesses  discussed  below,  system 
integrity,  risk  management,  and  configuration  management, 
occasionally  lack  discipline  in  their  application  during  the 
development  and  sustainment  of  systems. 

System  Integrity  Subprocess 

The  system-level  integrity  program  is  crucial  to  the  engineering 
and  management  process  used  to  ensure  the  design/modification, 
manufacture,  quality,  and  maintenance  of  a system  is  consistent 
and  compatible  with  both  its  intended  and  actual  use.  This  is 
necessary  to  ensure  the  required  levels  of  safety  and  reliability 
are  achieved  while  meeting  other  performance  requirements. 
When  technological  capabilities  are  incapable  of  maintaining 
sufficient  safety  margins  while  meeting  required  performance 
levels,  safety  margins  shall  be  maintained  via  the  modification 
of  inspection,  repair,  and/or  replacement  intervals  based  on  the 
life  used  and  margin  remaining.  This  necessitates  a feedback  or 
tracking  mechanism  to  ensure  adaptability  to  changes  in  usage. 
Additional  feedback  should  also  be  provided  to  ensure  the  cor- 
rectness and  completeness  of  technical  orders  and  training. 
Strong  ties  are  required  between  configuration  management  and 
the  system  level  integrity  program  to  allow  determination  of  the 
item’s  life  remaining  based  upon  how  the  system  is  used  and 
maintained  versus  its  expected  usage  at  the  beginning  of  the 
program. 

Throughout  the  development  (or  modification)  process,  several 
integrity  program  functions  should  be  performed.  Figure  1 
depicts  key  elements  of  the  system  integrity  process.  Each  ele- 
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ment,  and  its  function  or  related  activity  necessary  for  ensuring 
integrity  of  the  system,  is  discussed  briefly. 


Field  data  system(s):  Field  data  systems  should  provide  serial 
number,  part  number,  manufacturer  number,  tail  number,  accu- 
mulated stress,  or  hours  as  needed  to  be  able  to  determine  life 
used  and  indicate  when  maintenance,  replacement,  or  inspec- 
tions are  called  for  to  maintain  appropriate  design  margins.  Field 
data  systems  should  accurately  reflect  the  impact  of  functional 
failures  and  maintenance  activities  on  meeting  operational  capa- 
bilities. Systems  should  minimize  the  need  for  input  by 
maintainers  and  pilots.  Notes  from  maintainers  and  pilots  should 
be  available  for  review.  Databases  containing  maintenance 
information  should  provide  direct  links  to  manufacturing  data- 
bases (failure  reporting,  analysis,  and  corrective  action  systems) 
and  to  warranty  tracking  systems  to  ensure  manufacturers  have 
the  latest  field  information.  Information  should  also  be  available 
for  the  chief  engineer  to  review  current  fielded  status  and  priori- 
tization of  fixes,  etc. 

Reliability  & maintainability  (R&M)  analysis:  Provides 
system  level  design  considerations  to  ensure  mission  and  hard- 
ware reliabilities  are  achievable  and  consistent  with  the  user’s 
needs.  Integrity  program  tasks  ensure  a sound  understanding  of 
the  environment  and  usage  as  well  as  the  consistency  and  level 
of  product  quality  to  be  accounted  for  in  design. 

Failure  modes  effects  criticality  analysis  (functional):  Func- 
tional failure  modes  effects  criticality  analysis  (FMECA)  is 
accomplished  at  the  system  level  and  should  include  functional 
failures  of  lower-level  subsystems  (both  internal  failures  and 
loss  of  input).  The  functional  FMECA  describes,  through  the 
database,  how  functional  failures  propagate  through  the  system 
as  well  as  the  eventual  effects  of  said  functional  failures.  These 
are  expressed  in  terms  of  safety  in  addition  to  the  diagnostic 
indicators  that  should  appear  and  the  actions  the  pilot  or  main- 
tainer  should  take  to  mitigate  the  problem. 

Subsystem  safety  hazard  analysis:  Safety  draws  initial  infor- 
mation from  FMECAs.  Unlike  FMECAs  the  subsystem  safety 
hazard  analysis  (SSHA)  evaluates  multiple  failure  scenarios. 
The  SSHA  can  result  in  designating  items  as  safety  critical  and 
safety  significant.  A safety  significant  item  or  system  is  one  that 
requires  multiple  failures  to  cause  the  loss  of  a function.  For 


example,  loss  of  flight  controls  on  the  new  systems  usually 
requires  loss  of  three  independent  channels:  in  such  a case  each 
channel  is  considered  safety  significant. 

Diagnostics  & health  management  (DHM):  DHM  information 
is  included  in  functional  FMECA(s).  Since  both  information 
pointing  at  the  loss  of  a function  and  the  cause  of  the  problem 
are  in  the  FMECA,  it  is  a good  source  for  DHM  analysis.  DHM 
can  further  use  the  FMECA(s)  to  develop  fault- filtering  algo- 
rithms to  provide  better  isolation  and  to  reduce  false  alarms. 
Where  the  loss  of  function  is  significant,  it  may  also  be  pre- 
sented to  the  pilot  as  an  integrated  caution  and  warning  (ICAW). 
These,  too,  are  usually  part  of  FMECA(s). 

FMECA  (hardware  or  software):  Hardware  or  software 
(Hdw/Sw)  level  FMECAs  are  generally  done  by  subcontractors 
and  are  based  on  a piece-part  (for  software  it  means  computer 
software  unit)  analysis  of  the  subsystem  in  question.  Hdw/Sw 
level  FMECAs  relate  individual  piece-part  failure  modes  to 
losses  of  functions.  The  functional  losses  expressed  in  Hdw/Sw 
level  FMECAs  should  be  the  same  as  those  functional  losses 
examined  in  the  functional  FMECA.  Standardized  database 
structures  would  help  relate  these  level  FMECAs  to  functional 
FMECAs  and  reduce  the  manpower  needed  to  accomplish 
FMECAs  in  general.  Hdw/Sw  level  FMECAs,  when  tied  to 
functional  FMECAs,  reveal  which  part  failures  result  in  the  loss 
of  a safety  critical  and/or  mission  critical  function. 

Parts  control:  Parts  control  provides  control  for  safety-critical 
parts  through  a serial  number  tracking  system.  Parts  identified  as 
safety  critical  through  the  FMECA,  and  those  also  identified  as 
safety  significant  (needs  multiple  failures)  via  the  subsystem 
safety  hazard  analysis  (SSHA),  may  require  special  checks 
during  manufacturing  as  well  as  in  field  usage.  The  parts  control 
system  should  be  capable  of  ensuring  that  the  parts  purchased 
are  consistent  with  the  intended  usage  and  environment  and 
have  quality  levels  as  good  as  or  better  than  the  parts  they 
replace.  Necessary  part  functional  tolerances  should  be 
evaluated  when  considering  a replacement  part.  Authority 
should  reside  under  parts  control  engineering  as  opposed  to 
being  relinquished  to  a purchasing  agent  who  is  not  under  engi- 
neering control. 

Quality  control:  The  assurance  of  quality  for  production  arti- 
cles is  critical  to  assurance  and  preservation  of  OSS&E.  Quality, 
for  the  purpose  of  this  discussion,  refers  to  the  engineering  of 
the  product  to  meet  user’s  needs  reliably  (design  quality)  and  the 
manufacturing  of  production  units  repeatedly  in  complete 
agreement  with  the  design  (production  quality).  To  assure  the 
quality  of  design,  and  to  assure  that  the  as-built  configuration 
matches  the  as-designed  configuration,  an  effective  quality 
system  should  be  in  place. 

Process  control:  In  order  to  assure  OSS&E  of  a product,  the 
capability  and  stability  of  the  manufacturing  processes  are 
extremely  important.  Therefore,  the  processes  need  to  be 
qualified  and  controlled  by  the  manufacturers  throughout  the  life 
of  a system.  It  would  be  possible  to  test  fully  each  unit  of 
product,  including  comprehensive  testing  necessary  for  the 
intrinsic  manufacturing  processes  subsequent  to  the  first  pro- 
duction unit  delivery.  However,  a more  economical  approach  is 
to  assure  that  follow-on  units  are  functionally  identical  to  the 
one  that  is  tested.  This  should  address  the  product  characteristics 
that  exert  some  influence  over  the  product's  OSS&E.  Manufac- 
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turers  should  consider  the  minimum  set  of  criteria  listed  below 
to  assure  OSS&E  of  a system  (for  noncomplex  or  COTS  items, 
verification  inspection  and  testing  may  be  sufficient).  A prime 
contractor  for  the  system  is  responsible  for  the  flow  down  of 
these  criteria  to  appropriate  suppliers. 

1)  Identify  product’s  key  characteristics  (at  the  form,  fit,  and 
function  level)  that  are  related  to  OSS&E. 

2)  Identify  appropriate  product  appraisal  methods  (inspec- 
tion/test) for  key  characteristics  and  report  results  of  those 
appraisals. 

3)  Identify  key  manufacturing  process  parameters  that  deter- 
mine integrity  of  the  product. 

4)  Identify  the  required  key  manufacturing  process  capabili- 
ties (e.g.,  process  capability  index,  such  as  Cpk)  and  match 
them  to  the  design  requirements. 

5)  Implement  controls  over  the  key  manufacturing  processes. 
Risk  Management 

Risk  Management  is  a key  element  in  the  disciplined  engineer- 
ing process  required  to  assure  OSS&E  and  is  an  essential 
component  in  the  Department  of  Defense’s  strategy  for  acquir- 
ing and  sustaining  mission-capable  weapon  systems  in  an 
environment  of  diminishing  resources.  A disciplined,  compre- 
hensive risk  management  structure  involves  the  early  and 
continuous  identification  of  critical  program  risks,  and  the 
establishment  and  monitoring  of  risk  handling  plans.  When 
properly  implemented,  an  effective  risk  management  program 
facilitates  identification  of  areas  that  require  special  attention 
and  supports  setting  realistic  and  executable  technical,  schedule, 
and  cost  objectives.  Integrated  Risk  Management  is  the  practice 
of  controlling  risks  (those  things  that  are  in  conflict  with 
achieving  program  objectives).  ASC  uses  the  integrated  risk 
management  (IRM)  process,  which  consists  of  four  essential 
elements:  planning,  assessing,  handling,  and  monitoring  risk.  It 
is  implemented  by  IPTs,  throughout  die  life  of  a program,  to 
focus  resources  on  the  areas  of  the  program  that  are  most  critical 
to  delivering  weapon  systems  that  meet  the  user’s  mission 
needs.  To  be  effective,  risk  management  should  be  a continuous, 
daily  activity  employed  from  cradle  to  grave. 

Configuration  Management 

An  effective  configuration  management  (CM)  program  is 
imperative  in  order  to  maintain  operational  safety,  suitability, 
and  effectiveness  of  Air  Force  weapon  systems.  CM  provides 
the  discipline,  control,  management  of  data,  and  access  to  accu- 
rate data  that  is  necessary  to  implement  the  systems  engineering 
process.  CM  principles  are  inherent  in  sound  business  practices 
to  develop,  integrate,  test,  acquire,  operate,  maintain,  logistically 
support,  and  dispose  of  a weapon  system.  These  practices  apply 
across  parts,  assemblies,  subsystems,  hardware,  software  and 
firmware,  and,  indeed,  all  modifications  to  weapon  systems. 

Operational  safety,  suitability,  and  effectiveness  are  associated 
with  a specific  system  or  end-item  configuration.  The  specific 
configuration  and  its  characteristics  should  be  defined  by  engi- 
neering data  at  all  times.  Therefore,  a robust  configuration 
management  process  should  be  used  to  establish  and  preserve 
operational  safety,  suitability,  and  effectiveness  baselines. 
Permanent  and  temporary  configuration  changes,  as  well  as  the 


use  of  nonconforming  material,  will  be  reviewed  and  approved 
prior  to  implementation  or  installation.  Delegation  of  specific 
configuration  management  authority  between  organizations 
should  be  formally  documented.  This  authority  includes  con- 
figuration management  responsibility  for  supply,  maintenance, 
and  user-  and  test-initiated  changes. 

Best  Practices 

In  developing  this  policy,  the  IPT  made  a conscious  effort  not  to 
limit  any  use  of  best  practices  currently  reaping  benefits  in  both 
the  civil  and  the  Government  sectors.  This  approach  lowers  both 
acquisition  and  sustainment  costs  significantly  and  allows 
industry  and  Government  to  share  benefits.  Some  of  the  best 
practices  are  briefly  presented  in  this  section. 

Commercial  Off  the  Shelf  (COTS)/Nondevelopmental  Items 
(NDI) 

The  DoD  places  enormous  emphasis  on  the  use  of  COTS  and 
NDI  for  several  reasons.  COTS  and  NDI  have  many  common 
attributes  that  reduce  the  total  ownership  cost  of  Government 
systems.  All  COTS  are  NDI,  but  not  all  NDI  are  COTS.  The 
NDI  could  be  a product  or  part  that  has  been  developed  for 
another  military  application  and  may  not  have  led  to  use  in  any 
commercial  application.  In  contrast  the  COTS  are  products  and 
items  that  are  developed  for  the  commercial  market  and  are 
readily  available  for  Government  applications.  In  either  case, 
there  is  no  development  effort  involved.  Both  reduce  acquisition 
cycle  times  for  fielding  a product,  and  the  procurement  costs  are 
significantly  reduced. 

As  a keynote  speaker  at  a conference  in  1998,  Mr.  Robert 
Spitzer,  Vice  President  of  Engineering  at  Boeing  Commercial 
Airplane  Group,  remarked  “Throughout  years  of  air  travel, 
safety  has  improved  with  the  development  of  new  technologies” 
[9].  He  cited  the  development  of  jet  engine  technology  as  an 
example  to  support  his  point.  Use  of  COTS  items  leverages  the 
technology  innovations  of  the  commercial  market  that  are  out- 
pacing DoD’s  ability  to  exploit  them.  This  improves  safety 
while  increasing  the  availability  of  products  from  the  commer- 
cial marketplace  that  satisfy  military  needs.  The  use  of  proven 
technologies  further  reduces  another  major  portion  of  the  cost; 
namely,  that  associated  with  an  item's  testing/qualification. 

COTS  items  cover  the  entire  spectrum  ranging  from  systems 
down  to  piece  parts.  Where  the  Air  Force  mission  is  similar  to 
the  commercial  sector,  it  affords  a tremendous  advantage  to  the 
Air  Force  to  buy,  in  an  expeditious  manner,  a commercial 
aircraft  that  has  already  been  developed,  tested,  and  certified  by 
the  FAA.  Such  acquisitions  result  in  tremendous  savings  to  the 
Air  Force.  The  use  of  COTS  at  an  equipment  or  part  level 
improves  the  supply  chain  management  posture  for  fielded 
systems.  It  increases  the  availability  of  equipment  and  systems 
for  the  war  fighters.  Additionally,  utilizing  commercial  invento- 
ries provides  the  Air  Force  opportunities  to  reduce 
infrastructures  that  are  required  to  support  those  weapon 
systems. 

The  use  of  COTS/NDI  also  allows  our  contractors  to  use  their 
production,  maintenance,  and  test  facilities  as  well  as  associated 
staffs  and  processes  for  both  commercial  and  Government  use. 
Dual  use  of  products,  equipment,  processes,  and  practices  by  the 
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civil  and  military  sectors,  where  it  makes  sense,  is  a “win-win” 
situation  for  the  industry  and  Government.  It  not  only  cuts  down 
the  cycle  time,  but  it  also  eliminates  duplication.  Further, 
COTS/NDI  has  a potential  for  minimizing  inefficiencies  and 
avoiding  the  waste  of  valuable  national  resources  if  the  appro- 
priate level  of  technical  assessments  and  analyses  are 
accomplished  prior  to  their  use. 

There  are  two  key  elements  of  OSS&E  that  should  be  consid- 
ered when  using  COTS/NDI.  The  first  element  is  an 
understanding  of  the  inherent  capability  of  the  COTS/NDI  so  as 
to  form  an  initial  OSS&E  baseline.  The  second  element  is  a 
thorough  understanding  of  the  operational  requirements  associ- 
ated with  its  intended  use  as  an  end  item  or  as  an  integrated  part 
of  a larger  platform.  Lack  of,  or  incomplete  knowledge  of  the 
inherent  capability  of  the  COTS/NDI  does  not  exempt  the  chief 
engineer  from  OSS&E  responsibility.  It  is  the  responsibility  of 
the  chief  engineer,  as  part  of  the  overall  acquisition  strategy,  to 
acquire  or  develop  the  key  product  characteristics,  including 
COTS/NDI,  necessary  to  form  the  basis  for  an  initial  OSS&E 
assurance  baseline. 

Performance  Based  Specifications 

Military  specifications  and  standards  reform  initiated  in  1995 
was  a key  aspect  of  DoD  acquisition  reform.  Any  military  speci- 
fication or  military  standard  which  contained  detailed  levels 
specifying  "how  to  design"  systems/items  was  considered  inap- 
propriate for  new  development  efforts.  Most  were  cancelled  and 
others  needed  to  be  rewritten. 

Back  in  the  1970s,  the  Air  Force  had  embarked  on  a standardi- 
zation effort  called  MIL-PRIME  to  ensure  that  the  Government 
provided  requirements  for  acquiring  aircraft  products  in  terms  of 
operational  performance.  Having  used  this  approach  to  acquire 
many  systems,  the  industry  and  other  services  within  the  DoD 
endorsed  this  idea. 

The  MIL-PRIME  concept  was  generally  acceptable,  but  the 
documents  needed  restructuring  and  editing.  MIL-PRIME 
documents  were  not  consistent  in  content,  and  their  level  of 
detail  varied. 

Under  the  purview  of  the  Joint  Aeronautical  Commander’s 
Group  (JACG),  therefore,  this  initiative  to  specify  performance 
requirements  was  revived  with  industry  participation  under  the 
name  "Joint  Service  Specification  Guides  (JSSG)"  [10].  The 
JSSGs  are  being  developed  by  the  JACG  Aviation  Engineering 
Board  (membership  includes  Industry,  Army,  Navy,  and  Air 
Force).  These  Guides  contain  the  current  best  available  guidance 
for  identifying  general  performance  requirements  for  aeronauti- 
cal systems  and  subsystems.  These  guides  are  fundamental  for 
preparing  specifications  for  performance-based  systems  and 
major  subsystems  and  airworthiness  certification  criteria.  This 
JSSG  approach  provides  the  contractors  clear  accountability  in 
designing  safe,  effective  systems  in  a most  economical  way. 

Form  Fit  & Function  fF3i 

The  concept  of  form  fit  & function  has  been  applied  selectively 
to  military  systems  for  at  least  the  past  three  decades  [11].  Its 
application  has  brought  mNed  success  for  many  reasons  One  of 
those  reasons  has  been  that,  until  recently,  commercial  tech- 
nologies lagged  the  technologies  needed  by  the  Government  by 


several  years.  Another  major  obstacle  has  been  that  commercial 
parts  and  equipment  were  not  suitable  for  military  use  mainly 
due  to  their  lower  standards  for  reliability. 

This  trend  has  now  reversed.  A large,  competitive,  commercial 
marketplace  exists  with  expanding  domains  of  application. 
Competition  is  driving  technology  innovations  and  variety  in 
functions  and  performance.  The  technology  innovations,  in  turn, 
have  vastly  improved  reliability  of  electronics  equipment  and 
parts;  and  the  trend  continues.  At  present,  the  commercial  tech- 
nology is  outpacing  DoD  technological  needs.  Parts  and 
equipment  that  provide  the  desired  functionality  and  perform- 
ance at  competitive  prices  are  readily  available  commercially.  In 
addition,  system  architectural  schemes  allow  replacement  of 
older  equipment  with  new-technology  equipment  using  the  F3 
approach.  This  is  especially  attractive  for  the  military  applica- 
tions in  which  system  life  spans  several  decades. 

The  OSS&E  policy  takes  these  considerations  into  account  and 
encourages  the  exploitation  of  the  F3  concept.  As  long  as  the 
usage  spectrum  and  the  environments  are  conducive  to  the  use 
of  F3  parts  or  equipment,  the  original  equipment  manufacturers 
are  given  the  flexibility  to  make  that  determination  without 
being  hindered  by  the  Government.  The  F3  concept  is  an  inte- 
gral part  of  several  cost-cutting  initiatives  being  pursued  jointly 
by  the  industry  and  the  Government.  A few  of  these  initiatives 
are  total  system  performance  responsibility  (TSPR),  flexible 
sustainment,  and  diminishing  manufacturing  resources. 

FAA  Certifications 

The  FAA  certification  methodology  has  been  developed  over  40 
years  and  is  accepted  world  wide  as  the  premier  method  of 
certifying  aircraft.  The  new  Air  Force  policy  takes  full  advan- 
tage of  the  FAA  methodology,  when  practical.  For  systems  with 
unique  Government  missions,  the  Air  Force  has  created  a meth- 
odology that  parallels  the  FAA  policy  construct.  Current  USAF 
commercial-derivative  aircraft  with  missions  similar  to  commer- 
cial operators  (e.g.,  C-20,  VC-25,  C-32,  C-37)  will  be  FAA 
certified.  The  USAF  hybrid  commercial-derivative  aircraft  (e.g., 
E-3,  E-4)  should  comply  with  FAA  standards  to  the  extent 
possible.  Use  of  commercial-derivative  aircraft  in  meeting  Air 
Force  missions  is  a special  case  and  usually  requires  adaptation 
of  the  Air  Force  airworthiness  methodology.  Obtaining  and 
maintaining  FAA  certification  for  the  above  cases  is  both  cost 
effective  and  the  preferred  method  of  assuring  airworthiness. 

Methodologies 

System  Safety 

System  safety  is  a vital  part  of  the  OSS&E  for  the  life  of  a sys- 
tem. The  objective  of  system  safety  is  to  achieve  an  acceptable 
level  of  mishap  risk  through  a systematic  approach  of  hazard 
analysis,  risk  assessment,  and  risk  management.  Current  Air 
Force  policy  details  responsibilities  for  program  managers  with 
regard  to  the  US  Air  Force  Mishap  Prevention  Program.  Specifi- 
cally, program  managers  responsible  for  the  development  or 
modification  of  a system  are  to  establish  and  maintain  a tailored 
system  safety  program  in  accordance  with  MIL-STD-882D  [12]. 
MIL-STD-882D  has  recently  been  revised  with  industry  partici- 
pation under  the  guidelines  of  acquisition  reform.  The  resulting 
document  requires  judicious  imposition  of  MIL-STD-882D. 
Only  section  4 of  the  standard  is  contractually  binding.  This 
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section  provides  for  (1)  documentation  of  the  system  safety 
approach,  (2)  identification  and  tracking  of  hazards  and  (3) 
acceptance  of  residual  risks  by  the  appropriate  authority. 

In  addition  to  establishing  and  maintaining  a system  safety 
program,  the  policy  requires  system  safety  groups  (SSGs)  to  be 
established  for  aircraft  programs  unless  waived  by  the  appropri- 
ate Government  office.  The  purpose  of  the  SSG  is  to  oversee  the 
system  safety  program  throughout  the  life  cycle  of  the  system 
and  to  document  the  mishap  risk  review  process.  The  Chief 
Engineer  is  a key  member  of  the  SSG.  The  SSG  is  to  be  chaired 
by  the  program  manager  or  the  deputy  program  manager,  and  is 
to  have  a charter  which  includes  representatives  from  the  system 
user  in  the  membership.  The  system  safety  policy  also  defines 
the  appropriate  levels  of  Government  authority  for  acceptance  of 
residual  mishap  risks.  This  methodology  is  extremely  important 
for  the  Government  to  maintain  the  cost  control  by  focusing  on 
eliminating  unacceptable  risks. 

Operational  Clearances  at  Program  Milestones 

The  operational  clearance  approach  is  an  orderly,  incremental, 
sequential  activity  that  leads  to  aircraft  certification.  This  incre- 
mental clearance  approach  establishes  a framework  of 
responsibility  and  accountability  for  establishing  and  maintain- 
ing the  operational  safety,  suitability,  and  effectiveness  of 
weapon  systems  throughout  their  life.  The  operational  clearance 
approach  can  be  applied  to  new  acquisitions,  to  modifications  to 
frelded  systems,  and  to  modified  commercial  items.  This 
approach  should  be  applied  any  time  changes  are  made  to  the 
approved  configuration  of  a system  or  end  item.  The  configura- 
tion control  board  should  not  give  approval  for  any  temporary  or 
permanent  modification  unless  this  approach  has  been  followed. 
It  is  comprised  of  two  parts.  One  part  deals  with  clearance  for 
individual  equipment  items,  and  the  second  part  deals  with  the 
aircraft  as  a whole. 


will  require  an  EOC  and,  depending  on  the  magnitude  of  the 
modification,  most  likely  recertification  of  aircraft. 

Figure  2 is  a notional  representation  of  the  equipment  opera- 
tional clearance  methodology.  It  depicts  progressive  levels  of 
design  maturity  as  equipment  moves  to  higher  levels  of  clear- 
ances. This  methodology  would  be  applied  to  hardware, 
software,  and  removable/replaceable  items  comprising  the  vari- 
ous component  items  of  the  total  system. 


—6  FLIGHT  (Fully  Qualified) 

? FLIGHT  (No  Temporary  Restrictions) 
— 4 FLIGHT  (Use  with  Limitations) 

—3  NON-FLIGHT  (Unrestricted  use) 
"3—  NON-FLIGHT  (Use  with  Limitations) 
LABORATORY  USE  ONLY 
—ft  NOT  CLEARED  FOR  USE 
UNDER  ANY  CONDITION 


Figure  2 . 

Notional  Equipment  Operational  Clearance  Methodology 


The  exit  criteria  for  each  level  should  be  established.  Clearance 
to  the  next  level  should  not  be  permitted  unless  the  established 
minimum  exit  criteria  for  the  current  level  has  been  met.  To 
control  the  length  of  this  paper,  exit  criteria  for  all  levels  are  not 
presented.  However,  to  illustrate  how  the  graduation  to  next 
level  of  clearance  takes  place,  level  0 exit  criteria  are  provided 
below  which  show  some  top-level  minimum  criteria  that  should 
be  satisfied  to  clear  a particular  item  to  the  next  level;  that  is,  to 
level  1. 


Equipment  operational  clearance  (EOC):  The  EOC  method- 
ology described  here  (or  one  similar)  is  a disciplined 
engineering  process  for  assuring  OSS&E  of  aircraft.  It  should  be 
recognized  that  products  that  are  safe,  suitable,  and  effective 
result  from  the  disciplined  application  of  multiple  technical 
processes.  Some  of  these  key  processes  have  been  presented 
above.  These  processes  embody  a number  of  functional  disci- 
plines, each  with  its  own  unique,  expert  knowledge  base. 
Criteria  are  extracted  from  the  expert  knowledge  base  of  several 
functional  disciplines  that  have  a direct  effect  on  the  achieve- 
ment of  OSS&E.  Adherence  to  these  minimum  criteria  may  not 
guarantee  safety,  suitability,  and  effectiveness;  however,  devia- 
tions from  these  criteria  dramatically  increase  the  likelihood  of 
unfavorable  results.  Additionally,  more  detailed  criteria  may 
apply  given  particular  programmatic  details.  A prime  contractor 
may  have  a more  detailed  description  of  methodology  to  accom- 
plish the  equipment  operational  clearances.  At  the  discretion  of 
the  prime  contractor,  suppliers  and  equipment  developers  may 
be  required  to  follow  that  specific  methodology. 

The  contractor  accomplishes  initial  clearance  of  the  equipment 
prior  to  approval  for  first  flight  by  the  Government’s  aircraft 
program  manager.  The  update  to  the  clearance  is  accomplished 
for  the  follow-on  clearances  throughout  the  development  activi- 
ties leading  to  the  final  airworthiness  certification  for 
operational  use.  Any  modification  to  the  equipment  that 
received  an  EOC  as  part  of  original  aircraft  certification  effort 


1)  Functional  requirements  are  properly  defined  and  allocated 

2)  Integrity  analyses  are  complete 

3)  FMECA  is  complete  (functional,  hardware/software) 

4)  Safety  hazard  analysis  is  complete 

5)  Interface  requirements  are  documented 

6)  Detailed  design  criteria  are  documented 

7)  Conformity  check  is  complete 

8)  Laboratory  operating  restrictions  and  limitations  are  docu- 
mented 

9)  Laboratory  test  planning  is  complete 

Air  vehicle  operational  clearance:  When  we  speak  of  the  air 
vehicle  as  a whole,  in  contrast  to  the  EOC  discussed  above,  we 
are  concerned  only  with  establishing  clearance  levels  4 to  6, 
where  flight  operations  are  involved.  As  an  example,  minimum 
criteria  to  consider  when  clearing  the  air  vehicle  to  level  4 are  as 
follows: 

1)  All  equipment  operational  flight  clearances  are  at  level  4 or 
higher 

2)  Flight  test  planning  is  complete 

3)  Aiicrew  and  maintenance  personnel  are  trained 

4)  Safety  hazard  analysis  is  complete 
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5)  Flight  and  maintenance  manuals  are  reviewed  for  adequacy 

6)  All  aircraft  operating  restrictions  or  limitations  are  identi- 
fied and  documented 

7)  All  problems  from  lower  level  testing  are  reviewed  and 
dispositioned 

8)  All  complaints  from  pilot  simulations  are  reviewed  and 
dispositioned 

9)  All  problems  from  functional  checks  are  reviewed  and 
dispositioned 

10)  Independent  review  team  review  has  been  conducted  (for 
air  vehicle  first  flight) 

11)  All  ground  operation  restrictions  and  limitations  are  identi- 
fied and  documented 

12)  Ground  verification  testing  , flutter  excitation,  structural 
coupling,  and  electromagnetic  interference  tests  are 
complete 

13)  Taxi  runs  are  complete 

Unless  the  minimum  level  4 criteria  are  met,  the  air  vehicle 
should  not  be  cleared  for  first  flight.  In  all  cases,  the  individual 
hardware,  software,  or  other  removable/replaceable  items  of  the 
air  vehicle  should  have  achieved  an  equivalent,  or  higher,  level 
of  clearance  to  that  sought  for  the  air  vehicle.  As  mentioned 
above,  the  prime  contractor,  responsible  for  the  development  of 
the  system,  is  also  responsible  for  ensuring  clearances. 
However,  in  addition  to  the  contractor's  clearances,  the  final 
approval  authority  for  the  conduct  of  first  flight  resides  with  the 
Government.  With  a large  investment  usually  in  billions  of 
dollars  and  the  fate  of  a program  at  stake,  a comprehensive  and 
complete  review  must  be  accomplished  to  determine  airworthi- 
ness of  the  air  vehicle.  To  conserve  resources,  it  is  only  prudent 
to  conduct  a joint  Government  and  industry  review.  Depending 
on  the  technical  complexity  of  the  program,  a review  conducted 
by  a team  of  senior  level  experts  from  industry,  Government  and 
academia  is  highly  recommended.  Their  recommendations  to  the 
appropriate  Government  official  should  form  the  basis  for 
determination  whether  the  air  vehicle  is  safe  and  suitable  for 
first  flight.  Again,  more  detailed  criteria  may  be  added  given 
particular  programmatic  details. 

Iterative  application:  Once  level  6 clearance  has  been  achieved 
for  the  air  vehicle/equipment,  any  proposed  change  to  the 
approved  configuration  baseline  should  enter  the  incremental 
clearance  process  at  the  lowest  level  and  earn  its  way  to  the  top. 
This  iterative  nature  of  the  process  application  is  what  necessi- 
tates strict  configuration  control  of  the  baseline  configuration  by 
the  SM  and  the  chief  engineer.  It  is  the  linchpin  in  successful 
life  cycle  management  of  OSS&E. 

Product  Acceptance  Criteria  (PAQ 

The  methodology  of  product  acceptance  becomes  of  utmost 
importance  in  establishing  and  maintaining  weapon  system 
operational  safety,  suitability,  and  effectiveness.  In  the 
performance-based  environment,  it  is  almost  certain  that 
production  products  delivered  to  the  customer  will  not  be  of  the 
same  physical  configuration  as  the  original  qualified  article. 
Proper  documentation,  an  audit  trail  of  qualifications,  and  a 
structured  methodology  for  product  development  could  avoid 
unnecessary  expensive  testing  costs  when  accepting  production 
products.  Under  the  purview  of  the  Joint  Aeronautical 


Commanders,  a "Performance  Based  Product  Definition  Guide" 
[13]  has  been  developed  which  addresses  PAC  methodology  in 
more  detail. 

A robust  systems  engineering  process  should  go  beyond  simply 
providing  a product  design  which  has  been  verified  through  the 
qualification  process  to  meet  stated  requirements.  Design 
owners  should  also  define  and  document  “design  intent,”  which 
captures  the  physical  and  functional  aspects  of  the  design  solu- 
tion that  are  key  to  its  successful  function.  In  addition,  design 
owners  should  quantify  the  amount  of  variation  of  these  “key 
characteristics”  which  is  allowable  in  order  for  the  product  to 
function  as  intended.  This  information  set  is  a subset  of  the  total 
product  definition  that  completely  defines  the  product  configu- 
ration and  the  processes  used  to  produce  it. 

Once  the  key  characteristics  and  limits  of  acceptable  variation 
are  defined,  the  design  owner  can  develop  product  acceptance 
criteria  (PAC)  linked  directly  to  these  characteristics,  which  are 
traceable  to  the  performance  based  requirements.  Note  that 
product  acceptance  criteria  may  take  many  forms:  from  physical 
measurement  and  inspection  of  hardware,  to  an  acceptance  test 
procedure,  to  statistical  process  control.  The  specifics  will  vary 
from  case  to  case,  but  a common  attribute  is  that  any  product 
that  meets  the  criteria  will  possess  the  necessary  functionality 
regardless  of  differences  in  physical  configuration.  This  scheme 
also  provides  a baseline  against  which  the  design  owner  can 
evaluate  the  acceptability  of  future  design  and  process  changes. 

The  essential  point  is  that  the  design  owners  should  have  the 
means  to  determine  the  adequacy  of  the  products  that  they 
deliver  to  their  customers.  And  the  customers  should,  in  turn, 
have  assurance  that  the  criteria  that  are  to  be  used  for  product 
acceptance  are  based  on  sound  engineering  practices  and  are 
linked  directly  to  the  system  performance  requirements.  This 
qualification  process  should  be  complete  at  the  end  of  the  devel- 
opment phase  so  that  the  PAC  are  available  for  use  during  the 
production  program  and  the  sustainment  phase. 


3.  CONCLUSION 

Mishaps  in  the  aircraft  business  are  inevitable.  There  are  many 
factors  that  contribute  to  mishaps  and  most  of  those  factors 
cannot  be  completely  controlled.  Therefore,  complete  prevention 
of  mishaps  is  an  impossible  task.  Safety  hazards  should  be 
identified  and  eliminated  or  reduced  to  acceptable  levels  of  risk 
over  the  operational  life  of  the  system.  And  those  risks  should  be 
managed  throughout  the  life  of  a system. 

A reality  that  aggravates  this  challenge,  today  and  for  the  fore- 
seeable future,  is  managing  risk  within  an  environment  of  DoD 
downsizing  and  scarce  budgets.  OSS&E  will  continue  to  be  a 
primary  concern  of  the  USAF.  The  process  must  continuously 
adapt  to  scarce  resources  in  order  to  deal  effectively  with  the 
changing  situations  in  the  world.  Partnerships  with  industry,  use 
of  available  proven  processes  and  methodologies,  and  use  of 
best  practices  are  steps  in  the  right  direction  to  cost-effective 
improvement  of  OSS&E  and  mishap  reduction. 

Issuance  of  a high-level  Air  Force  policy  is  only  the  first  step 
toward  this  endeavor.  The  policy  must  be  effectively  promul- 
gated throughout  the  Air  Force  for  the  entire  product  line.  It  is 
incumbent  upon  program  managers,  chief  engineers  and  their 


staffs,  maintained,  and  operators  to  maintain  a diligent  approach 
to  this  serious  matter.  Concerned  parties  must  understand  their 
specific  roles  and  responsibilities  and  must  work  in  partnership 
with  industry  as  a cohesive  team  to  meet  this  challenge.  The 
chief  engineer  is  the  critical  link  between  the  contractor  and  the 
customers.  His  task  is  to  ensure  implementation  of  a disciplined 
engineering  approach  that  establishes  and  preserves  technical 
integrity  throughout  the  life  of  Air  Force  systems.  Yet,  the 
degree  of  success  depends  on  the  level  of  cooperation  that 
stakeholders  give  to  the  chief  engineer.  Collaboration  potentially 
can  assure  OSS&E  of  USAF  systems,  which  furthermore 
extends  to  reducing  the  DoD  economic  burden. 
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